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ABSTRACT 



In 2001, the National Assessment of Educational Progress 
(NAEP) conducted a geography assessment of the nation's fourth-, eighth-, and 
twelfth-grade students. This report presents the results of that assessment. 
Results in 2001 are compared to results of 1994 ' s NAEP geography assessment, 
which was the preceding NAEP geography assessment and the only other 
geography assessment conducted under the current framework. Students' 
performance on the assessment is described in terms of average scores on a 
0-500 scale and of percentage of students attaining three achievement levels: 
(1) basic; (2) proficient; and (3) advanced. Average geography scores for 
fourth and eighth graders were higher in 2001 than in 1994, while the 
performance of twelfth graders was not significantly different. At both 
grades 4 and 8, score increases occurred among the lower-performing students. 
The 2001 assessment showed that 21% of fourth graders, 30% of eighth graders, 
and 25% of twelfth graders performed at or above the proficient level for 
their respective grades. These levels are identified as those at which all 
students should perform. Both grades 4 and 8 showed an increase from 1994 to 
2001 in the percentage of students at or above basic. No significant changes 
occurred in the percentage at or above "Proficient" at any grade. In addition 
to overall results, the report provides data on the performance of various 
subgroups of students and information about the contexts for learning by 
administering questionnaires to assessed students, their teachers, and their 
school administrators. The report contains six chapters, each of which 
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What is The Nation’s Report Card? 

THE NATION’S REPORT CARD, the National Assessment of Educational Progress (NAEP),is the only nationally 
representative and continuing assessment of what America’s students know and can do in various subject areas. Since 
1969, assessments have been conducted periodically in reading, mathematics, science, writing, history, geography, and 
other fields. By making objective information on student performance available to policymakers at the national, state, 
and local levels, NAEP is an integral part of our nation’s evaluation of the condition and progress of education. Only 
information related to academic achievement is collected under this program. NAEP guarantees the privacy of 
individual students and their families. 

NAEP is a congressionally mandated project of the National Center for Education Statistics, the U.S. Department 
of Education. The Commissioner of Education Statistics is responsible, by law, for carrying out the NAEP project 
through competitive awards to qualified organizations. NAEP reports directly to the Commissioner, who is also 
responsible for providing continuing reviews, including validation studies and solicitation of public comment, on 
NAEP’s conduct and usefulness. 

In 1988, Congress established the National Assessment Governing Board (NAGB) to formulate policy guidelines 
for NAEP. The Board is responsible for selecting the subject areas to be assessed from among those included in the 
National Education Goals; for setting appropriate student performance levels; for developing assessment objectives and 
test specifications through a national consensus approach; for designing the assessment methodology; for developing 
guidelines for reporting and disseminating NAEP results; for developing standards and procedures for interstate, 
regional, and national comparisons; for determining the appropriateness of test items and ensuring they are free from 
bias; and for taking actions to improve the form and use of the National Assessment. 
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ixecutive Summary 



The National Assessment of Education al Progress (NAEP) is 
the nation’s only ongoing representative sample survey of 
student achievement in core subject areas. In 2001, NAEP 
conducted a geography assessment of the nation’s fourth-, 
eighth-, and twelfth-grade students. 

Authorized by Congress and administered by the National 
Center for Education Statistics (NCES) in the U.S. 
Department of Education, NAEP regularly reports to the 
public on the educational progress of students in grades 4, 8, 
and 12. This report presents the results of the NAEP 2001 
geography assessment for the nation. Results in 2001 are 
compared to results of the 1994 NAEP geography 
assessment, which was the preceding NAEP geography 
assessment and the only other geography assessment 
conducted under the current framework. Students’ 
performance on the assessment is described in terms of 
average scores on a 0—500 scale and in terms of the 
percentage of students attaining three achievement levels: 
Basic, Proficient, and Advanced. The achievement levels are 
performance standards adopted by the National Assessment 
Governing Board (NAGB) as part of its statutory 
responsibilities. They represent collective judgments of what 
students should know and be able to do. 
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As provided by law, the Deputy Com- 
missioner of Education Statistics, upon 
review of a congressionally mandated 
evaluation of NAEP, determined that the 
achievement levels are to be used on a trial 
basis and should be interpreted with 
caution. However, both the Deputy Com- 
missioner and the NAGB believe these 
performance standards are useful for under- 
standing trends in student achievement. 
They have been widely used by national 
and state officials as a common yardstick of 
academic performance. 

In addition to providing average scores 
and achievement-level performance in 
geography for the nation s fourth-, eighth-, 
and twelfth-graders, this report provides 
results for subgroups of students at those 
grade levels defined by various background 
characteristics (such as gender, race/ 
ethnicity, region, parents’ education, etc.) 
and classroom contexts for learning. A 
summary of major findings from the 2001 
NAEP geography assessment is presented 
on the following pages. Differences be- 
tween results across years or between 
groups of students are discussed only if they 
have been determined to be statistically 
significant. Readers are cautioned that the 
relationship between a contextual variable 
and student performance is not necessarily 
causal. 

Major Findings at 
Grades 4, 8, and 12 

□ Average geography scores for fourth- 
and eighth-graders were higher in 2001 
than in 1994, while the performance of 
twelfth-graders was not significantly 
different. 

□ At both grades 4 and 8, score increases 
occurred among the lower-performing 
students (at the 10th and 25th percentiles). 



□ The 2001 geography assessment showed 
that 21 percent of fourth-graders, 30 
percent of eighth-graders, and 25 per- 
cent of twelfth-graders performed at or 
above the Proficient level for their 
respective grades. These levels are 
identified by NAGB as those at which 
all students should perform. 

□ Both grades 4 and 8 showed an increase 
from 1994 to 2001 in the percentage of 
students at or above Basic. There were no 
significant changes in the percentage at 
or above Proficient at any grade. 

Results for Student Subgroups 

In addition to overall results, NAEP reports 
on the performance of various subgroups 
of students. Observed differences between 
student subgroups in NAEP geography 
performance may reflect a range of socio- 
economic and educational factors not 
addressed in this report or by NAEP. 

Gender 

□ There was no statistically significant 
change at any grade in the average scores 
of either male or female students be- 
tween 1994 and 2001. 

□ In 2001 as in 1994, male students at 
grades 4, 8, and 12 had higher average 
scores than female students. 

Race/Ethnicity 

□ At grade 4, Black students had higher 
average scores in 2001 than in 1994. 

□ In 2001, White, Asian/Pacific Islander, 
and American Indian students had 
higher average scores than Black and 
Hispanic students at all three grades. 

□ The 2001 results show a narrowing of 
the average score point difference be- 
tween White students and Black students 
at grade 4. 
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Region of the Country 

□ Between 1994 and 2001, the average 
scores of fourth-graders increased in the 
Northeast, and the average scores of 
eighth-graders increased in the Southeast. 

□ Fourth- and eighth-grade students in the 
Northeast and Central regions outper- 
formed students in the West in 2001, and 
students in the Central region also 
outperformed their counterparts in the 
Southeast. Twelfth-graders in the Central 
region had higher average scores than 
twelfth-graders in the Southeast. 

Parents’ Highest Level of Education 

□ Twelfth-graders whose parents had not 
graduated from high school had higher 
average scores in 2001 than in 1994. 

□ The higher the parental education level 
reported, the higher the average score 
attained by students at both grades 8 and 
12 in 2001. 

Type of School 

□ Eighth-grade public school students had 
higher average scores in 2001 than in 
1994. 

□ In 2001, nonpublic school students 
outperformed public school students at 
all three grades. 

□ In 2001, Catholic school students out- 
performed public school students at 
grades 4, 8, and 12. Apparent differences 
between public school and other 
nonpublic school students were not 
statistically significant. 

Type of Location 

□ In 2001, students in rural and urban 
fringe locations had higher average 
scores than central city students at grades 
4, 8, and 12. 



Eligibility for Free/Reduced-Price 
Lunch 

□ At every grade in 2001, the average 
score for students who were eligible for 
the Free/Reduced-Price School Lunch 
program was lower than the average for 
students who were not eligible for the 
program (i.e., those above the poverty 
guidelines). 

Classroom Contexts for Learning 

NAEP collects information about the 
contexts for student learning by adminis- 
tering questionnaires to assessed students, 
their teachers, and their school administra- 
tors. Using the student as the unit of 
analysis, NAEP examines the relationship 
between selected contextual variables 
drawn from these questionnaires and 
students’ average scores on the geography 
assessment. 

Teacher Preparation 

□ Ninety-three percent of fourth-grade 
students had teachers who indicated 
their graduate/undergraduate major or 
minor was elementary education, and 
about one-quarter (28 percent) of 
eighth-grade students had teachers who 
indicated they had a graduate/under- 
graduate major or minor in geography 
or geography education. 

□ A higher percentage of fourth-grade 
students in 2001 had teachers who 
reported they were very prepared to 
teach geography than did students in 
1994. Forty-four percent of eighth-grade 
students in 2001 had teachers who 
reported they were very prepared to 
teach geography. 
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Geography Skills Taught 

□ The percentage of eighth-grade students 
who studied maps and globes at least 
once or twice a week increased in 2001 
as compared to 1994. 

□ There was an increase in the percentage 
of eighth- and twelfth-grade students 
who studied natural resources once or 
twice a week in 2001 as compared 
with 1994. 

□ The percentages of eighth-grade stu- 
dents who studied countries and cultures 
in their geography instruction at least 
once or twice a week were greater in 
2001 than in 1994. 

Geography Course-Taking 

□ A higher percentage of eighth-graders in 
2001 reported taking geography in sixth, 
seventh, and eighth grades than did their 
counterparts in 1994. 

□ The percentage of twelfth-grade 
students taking geography courses at 
each grade level during their high 
school years increased in 2001 from 
the percentage reported in 1994. 

□ In 2001 at grade 8, students who re- 
ported taking two or three years of 
geography had higher scores than those 
who took it for fewer years. Twelfth- 
graders who reported taking one year or 
less of geography had higher average 
scores than those who took 3 or 4 years 
of geography. 

Use of Computers 

□ Students at grades 4, 8, and 12 who used 
the Internet or CD-ROM materials to a 
small or moderate extent had higher 
scores than students who did not use 
these tools at all. 



Becoming a More Inclusive NAEP 

In the 2001 geography assessment, the 
NAEP program used a split-sample design, 
so that trends in students’ geography 
achievement could be reported across 
assessment years and, at the same time, the 
program could continue to examine the 
effects of including special-needs students 
assessed with accommodations. Included in 
this report is an overview of the second set 
of results that include special-needs stu- 
dents who required and were provided 
accommodations during the assessment 
administration. 

□ In the sample where accommodations 
were not permitted, between 44 and 48 
percent of the special-needs students at 
each of the three grade levels (between 
5 and 8 percent of all students) were 
excluded from NAEP testing by their 
schools. In the sample where accommo- 
dations were offered, between 23 and 
24 percent of the special-needs students 
were excluded from the assessment 
(between 2 and 4 percent of the total 
sample). 

□ At grade 8, the average score when 
accommodations were permitted was 
lower than the average score when 
accommodations were not permitted. 

At grades 4 and 12, there were no 
statistically significant differences be- 
tween the average scores of students 
when accommodations were permitted 
and when accommodations were not 
permitted. 
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NAEP 2001 Geography Assessment 




What is the NAEP 

geography 

assessment? 

How does the 
NAEP geography 
assessment 
measure and 
report student 
progress? 



Introduction 

After more than 50 years during which geography was 
largely replaced by social studies in American public schools, 
geography education began to experience a revival during 
the 1980s and 1990s. 1 Contributing to the change was a 
growing belief in the relevance of geography to addressing 
economic, political, and environmental issues at the 
national and global level. Moreover, geography 
education was increasingly seen as an essential tool in 
the creation of effective citizens. This process gained 
momentum through the work of various 
organizations concerned with geography and 
geography education. These groups encouraged a 
more positive attitude toward geography and 
provided important guidance for reestablishing 
geography in the school curriculum. 2 Two surveys of 
geographic literacy, in 1988 and 1994, provided 
statistical evidence that student knowledge and skills 
fell far short of what was needed for responsible 
citizenship. 3 By the end of 1990, Congress had authorized 
development of a broad-based National Assessment of 
Educational Progress (NAEP) geography assessment at 



1 Salter, C. L. (1990). Missing the magic carpet: The real significance of geographic ignorance. 
Princeton, NJ: Educational Testing Service. 

2 Joint Committee on Geographic Education. (1984). Guidelines for geographic education : 
Elementary and secondary schools. Washington, DC: Association of American Geographers 
and the National Council for Geographic Education. 

3 Allen, R., Bettis, N., Kurfman, D., MacDonald, W., Mullis, I.V. S., & Salter, C. (1990). 
The geography learning of high school seniors . Princeton, NJ: National Assessment of 
Educational Progress, Educational Testing Service. 

Persky, H. R., Reese, C. M., O’Sullivan, C.Y., Lazer, S , Moore, J. D., & Shakrani, S. 
(1996). NAEP 1994 geography report card. Washington. DC: National Center for 
Education Statistics, Office of Educational Research and Improvement, 

U.S. Department of Education. 
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grades 4, 8, and 12, and the President and 
nation’s governors had declared geography 
to be one of five core subjects in their 
National Education Goals. 

Progress toward increasing the promi- 
nence of geography in the elementary and 
secondary school curriculum has generally 
been good. The 1990s saw the publication 
of the Geography Framework for the 1994 
National Assessment of Educational Progress 
and the NAEP geography assessment in 
1994, the introduction of the National 
Geography Standards, and the institution of 
the National Geographic Alliance Net- 
work. 4 The alliance is a professional orga- 
nization encouraged and supported with 
grants from the National Geographic 
Society Education Foundation. Geo- 
graphic Alliances are present in all 50 states, 
and are comprised of primary, secondary, 
community college, and university geogra- 
phy educators interested in the enhance- 
ment of geography education. The number 
of states with geography standards has been 
increasing steadily as well. According to 
recent data collected by the National 
Geographic Society, 48 states plus the 
District of Columbia now have geography 
standards in place, 37 of which are based 
on the National Geography Standards. 
However, only 13 states require a geogra- 
phy course as a requirement for high 
school graduation. Moreover, in 27 states 
geography is not tested in mandated state 
examinations, while in some other states 



the portion of mandated tests devoted to 
geography is very small. As a result, there 
could be little incentive for teachers to 
emphasize geography instruction when 
higher stakes are attached to other subjects. 5 
The results from the 2001 NAEP geography 
assessment provide policymakers, educators, 
and the general public with a new, objective 
tool with which to evaluate the country’s 
progress toward geographic literacy. 

Overview of the 2001 
National Assessment of 
Educational Progress 

For over 30 years, the National Assessment 
of Educational Progress (NAEP) has been 
authorized by Congress to collect, analyze, 
and report reliable and valid information 
about what American students know and 
can do in core subject areas. NAEP assesses 
the performance of public and nonpublic 
school students in grades 4, 8, and 12. In 
2001, student performance in geography 
and U.S. history was assessed at all three 
grades. This report deals only with the 
results of the geography assessment. 

All NAEP assessments are based on 
content frameworks developed through a 
national consensus process. The NAEP 
2001 geography assessment was the second 
administration of an assessment based on 
the Geography Framework for the 1994 
National Assessment of Educational Progress , 
which was originally developed for the 
1994 assessment. 6 In both 1994 and 2001, 



4 National Assessment Governing Board. (1994). Geography framework for the 1994 National Assessment of Educational 
Progress. Washington, DC: Author. 

Geography Education Standards Project. (1994). Geography for life: National geography standards. Washington, DC: 
National Geographic Research and Exploration. 

5 Munroe, S. and Smith, T. (1998). State geography standards. Fordham Report, 2(2), http://www.edexcellence.net/ 
standards/geography/geograph .htm . 

Dean, A. (2002). Unpublished data. National Geographic Education Foundation. 

Council of Chief State School Officers. (2000). Key state education policies on K-12 education: 2000. Washington, 
DC: Author. 

6 National Assessment Governing Board. (1994). Geography framework for the 1994 National Assessment of Educational 
Progress . Wa sh i ngt o n , D C : A ut h or. 
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assessments based on the framework were 
administered to national samples of fourth-, 
eighth-, and twelfth-graders. 

This report describes the results of the 
2001 geography assessment at grades 4, 8, 
and 12 and compares results in 2001 to 
those in 1994. Comparisons across assess- 
ment years are possible because the assess- 
ments were developed under the same 
basic framework and share a common set 
of geography questions. In addition, the 
populations of students were sampled and 
assessed using comparable procedures. 

The Geography Framework 

Although NAEP had conducted a geogra- 
phy assessment at grade 12 in 1988, a more 
comprehensive NAEP geography frame- 
work was developed for the 1994 assess- 
ment. The new framework provided the 
operational specifications for both the 1994 
and 2001 assessments. The development of 
the framework was managed by the Coun- 
cil of Chief State School Officers 
(CCSSO) and adopted by the National 
Assessment Governing Board (NAGB). 
Approximately 50 professional geographers, 
educators, administrators, and other inter- 
ested individuals worked to achieve con- 



sensus on the general goals as well as the 
specific language of the framework. In 
addition, several hundred educational 
experts and interested members of the 
public contributed to the process, either by 
participating in public hearings or by 
reviewing drafts. The framework document 
produced by this consensus process called 
for the assessment of a broad range of 
outcomes. It represented an ambitious 
vision both of what students should know 
and be able to do in geography, and of the 
ways in which those competencies should 
be tested. 

The geography framework is organized 
along two dimensions, a content dimension 
and a cognitive dimension. The content 
dimension forms the heart of the frame- 
work. It is divided into three main content 
areas covering the breadth of geography 
learning outcomes — knowledge and 
skills — that would flow from good geogra- 
phy instruction. 

The geography framework specifies the 
percentage of assessment time to be 
devoted to each content area. Figure 1.1 
shows how the assessment time is distrib- 
uted for each of the three grades: 40 




SOURCE: National Assessment Governing Board, Geography Framework for the 2001 National Assessment of Educational Progress. 
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percent of assessment time goes to Space 
and Place, and 30 percent each to Environ- 
ment and Society and to Spatial Dynamics 
and Connections. The percentages are 
important both because they guide the 



development of test questions and because 
they determine how much weight each 
content area receives in computing overall 
test scores. Figure 1.2 provides descriptions 
of each content area. 



Figure 1.2 




Content Area 
Descriptions 





Space and Place: 

Knowledge of geography as it relates to particular places on Earth , to spatial patterns on Earth's surface , 
and to physical and human processes that shape such spatial patterns. 

Space is the basic resource and organizing element for geography. Patterns that are illustrated 
on maps reflect both natural features and human activities. This content area requires students 
to distinguish between and understand the spatial distribution of physical and human charac- 
teristics. Students must locate significant features and places on Earth, recognize existing 
patterns in the distribution of features and places, and comprehend the reasons for the 
development and existence of these patterns. 



Environment and Society: 

Knowledge of geography as it relates to the interactions between environment and society : 

Geography is an integrative discipline that focuses on the interrelationships between the physical 
environment and society. Human adaptation to and modification of the environment have 
economic and political implications. Understanding the nature, scale, and ramifications of such 
environmental transformations is fundamental in geography education, and isthecore ofthis 
content area. Students must be aware that every environmental issue lends itself to many 
interpretations, depending on the people's perspectives. Students must consider such multiple 
perspectives as they evaluate decisions about issues, such as land use and resource develop- 
ment, because the results of such decisions often have complicated and unpredictable conse- 
quences. Learning to make wise decisions concerning the costs and benefits of environmental 
modification is an expressed goal of geography education. 



Spatial Dynamics and Connections: 

Knowledge of geography as it relates to spatial connections among people, places ; and regions. 

This content area explores critical problems in human interaction. It requires students to 
demonstrate comprehension of cultural, economic, and political regions and the connections 
among them. Students must understand how peoples and places are alike and how they differ. 
They should know that people of every country and every nation are increasingly connected to and 
dependent upon other peoples and places of the world for both human and natural resources. In 
this content area, students must demonstrate the knowledge that the world's resources are 
unevenly distributed, and an understanding of how this contributes to the movements of people, 
patterns of trade, and conflict. 



SOURCE: National Assessment Governing Board, Geography Framework for the 1994 and 2001 National Assessment of Educational Progress. 





CHAPTER t 



GEOGRAPHY REPORT CARD 




Three cognitive areas or levels comprise 
the cognitive dimension of the geography 
assessment. The framework labels them as 
Knowing, Understanding, and Applying, 
and defines them as follows. 

Knowing-What is it? Where is it? 

In this area, students are assessed on their 
ability to perform two related functions 
with respect to information: a) an observa- 
tion function and b) a recall function. 
Students should be able to observe differ- 
ent elements of the landscape and answer 
questions by recalling, for example, the 
name of a place or a resource indigenous to 
a particular country or by finding informa- 
tion about trading patterns among several 
countries. 

Understanding — Why is it there? How did 
it get there? What is its significance? 

In this area, students attribute meaning 
to what has been observed and explain 
events. Putting events in context and 
explaining them requires students to see 
connections among diverse pieces of 
geographic information and to use that 
information to explain existing patterns 
and processes on Earth. 




Distribution of geography assessment time across cognitive areas, grades 4, 8, and 12: 2001 





Knowing 


Understanding 


Applying 


Grade 4 


45 % 


30 % 


25 % 


Grade 8 


40 % 


30 % 


30 % 


Grade 12 


30 % 


30 % 


40 % 



Applying — How can knowledge and 
understanding be used to solve geographic 
problems? 

Applying geography knowledge and 
understanding requires a range of higher- 
order thinking skills. Students classify, 
hypothesize, use inductive and deductive 
reasoning, and form problem-solving 
models. They use many tools and skills of 
geography as they attempt to develop a 
comprehensive understanding en route to 
proposing viable solutions. 

Student performance in the three cogni- 
tive areas was not reported on separate 
subscales. Rather, the three areas were used 
to help guide development of the assess- 
ment instrument. The percentages of 
assessment time to be devoted to each 
cognitive area, as specified in the frame- 
workfare displayed in table 1.1. 

Together the content and cognitive 
dimensions of the assessment form a matrix 
in which each content area is measured at 
each cognitive level. 



SOURCE: National Assessment Governing Board, Geography Framework for the 1994 and 2001 National Assessment of Educational Progress. 
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Geography Assessment 
Instruments 

As the only federally authorized ongoing 
assessment of geography achievement, 
NAEP must reflect the spirit of the frame- 
work as well as the specifications provided 
by it. In order to achieve those goals, the 
assessment development process involved 
stages of review by measurement experts 
and a committee of teachers, teacher 
educators, and curriculum specialists expert 
in geography All components of the 
assessment were evaluated for curricular 
relevance, developmental appropriateness, 
and fairness. The National Assessment 
Governing Board (NAGB) gave final 
approval for NAEP test questions. A list of 
the geography development committee 
members for the 2001 assessment is pro- 
vided in appendix C. 

The 2001 geography assessment booklets 
at grades 4, 8, and 12 contained either three 
or four sections: a set of general background 
questions, a set of subject-related back- 
ground questions, and one or two sets, or 
“blocks,” of cognitive questions assessing 
knowledge and skills in geography. The 
general background questions are used to 
collect some important basic information 
about students. These questions tend to 
remain fairly constant across different 
NAEP assessments. The subject-related 
questions are designed for specific assess- 
ments or for assessments given in an indi- 
vidual year. The questions in the geography 
assessment asked students to give informa- 
tion about their school practices, such as 
the frequency with which they used the 
Internet or a CD-ROM to study geogra- 
phy, how often they received instruction in 
using maps and globes, and when they had 



taken a geography course. All students 
participating in the geography assessment at 
a particular grade received the same back- 
ground questions. 

The geography assessment as a whole 
contained 91 questions at grade 4, 124 
questions at grade 8, and 123 questions at 
grade 12. The grade 4 assessment was 
divided into six 25-minute blocks, while 
both the grade 8 and grade 12 assessments 
contained nine blocks, eight of which were 
25-minute blocks and one of which was a 
50-minute block. However, to reduce the 
burden on individuals, each student an- 
swered only a small portion of the total 
number of questions — ^either two 25- 
minute blocks or one 50-minute block. 

The 50-minute blocks administered at 
grades 8 and 12 focused on a particular 
geographic topic. In addition, one block at 
each grade was based entirely upon a 
student atlas that was provided to students. 
The assessment time for each grade, there- 
fore, was 50 minutes plus the 10-15 min- 
utes needed to complete the background 
questions. 

Each block of geography questions 
consisted of both multiple-choice and 
“constructed-response” questions. (“Con- 
structed response” is the term used to 
describe test questions in which students 
produce their own response, as distinct 
from multiple-choice questions, in which 
students choose an answer from one of 
several options.) Typically, a block will 
contain about 16-18 questions, but there is 
considerable variation depending on the 
balance between multiple-choice and 
constructed-response questions. Overall, 
more than 50 percent of student assessment 
time was devoted to the latter question 
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type. In addition, of the time reserved for 
constructed-response questions approxi- 
mately 20 percent was used for “produc- 
tion” questions in which students engaged 
in such tasks as indicating place locations 
on outline maps, drawing routes between 
points on a map, and drawing maps and 
diagrams based upon written descriptions. 
Two types of constructed-response ques- 
tions were used: 

□ short-constructed-response questions 
that required students to provide brief 
written answers of one or two sentences 
or complete a limited production task; 
and 

□ extended-constructed-response ques- 
tions that required students to provide 
answers of a paragraph or more in length 
or engage in an extensive production 
task like producing a map. 

Examples of multiple-choice, short- and 
extended-constructed-response and pro- 
duction questions are provided in chapter 
6. Additional information about the design 
of the 2001 geography assessment is pre- 
sented in appendix A. 

Description of School 
and Student Samples 

The NAEP 2001 geography assessment 
included representative samples of both 
public and nonpublic schools. For the 
reporting sample, approximately 7,000 
fourth-graders, 9,000 eighth-graders, and 
9,000 twelfth-graders were assessed. The 
number of schools in the reporting sample 
were 365 at fourth grade, 369 at eighth 
grade, and 374 at twelfth grade. Each 
selected school that participated in the 
assessment and each student assessed 
represent a portion of the population of 
interest. For additional information on 



sample sizes and participation rates, see 
appendix A. 

This report contains two different sets of 
national results based on two reporting 
samples that differed in terms of whether 
or not accommodations were made avail- 
able to special-needs students. The national 
results presented in chapters 2, 3, 4, and 6 
of this report are based on a nationally 
representative sample that included special- 
needs students only if they could be as- . 
sessed meaningfully without accommoda- 
tions. These results can be compared to 
those from 1994 because accommodations 
were also not made available in that assess- 
ment year. Chapter 5 presents a second set 
of national results from 2001 for a repre- 
sentative sample that includes the perfor- 
mance of students who required and were 
provided with accommodations (e.g., 
bilingual dictionary, extended time, small 
group testing). No comparison of these 
results to those from 1994 can be made 
because of the inclusion of these accom- 
modated special-needs students. 

In the sample that did not permit ac- 
commodations, 8 percent of fourth-graders, 
8 percent of eighth-graders, and 5 percent 
of twelfth-graders were excluded from the 
geography assessment in 2001. School staff 
familiar with these students made the 
determination, based upon NAEP s inclu- 
sion criteria, that these students could not 
be assessed meaningfully without accom- 
modations because of their disability and/ 
or limited English proficiency. In 1994, 5 
percent at both the fourth- and eighth- 
grades, and 3 percent at the twelfth-grade 
were excluded. Additional information 
regarding exclusion rates is provided in 
appendix A. 
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Reporting the 
Assessment Results 

Student performance on the NAEP geog- 
raphy assessment is presented in two ways: 
as average scores on the NAEP geography 
scale, and in terms of the percentage of 
students attaining NAEP geography 
achievement levels. The average scale 
scores are a measure of what students know 
and can do in geography. The achievement- 
level results indicate the degree to which 
students’ performance meets expectations of 
what they should know and be able to do. 

Average scale score results are presented 
on the NAEP geography composite scale, 
which ranges from 0—500. Students’ re- 
sponses on the NAEP 2001 geography 
assessment were analyzed to determine the 
percentages of students that responded 
correctly to each multiple-choice question 
and the percentages of students that re- 
sponded at each score level for the con- 
structed-response questions. Scales that 
summarize results for each of the three 
content areas described earlier were cre- 
ated. The composite scale is a weighted 
average of the separate subscales for the 
three content areas. The weight for each 
content area corresponds to its relative 
importance as prescribed in the NAEP 
geography framework. A full description of 
NAEP scale procedures can be found in 
the forthcoming NAEP 2001 Technical 
Report. 

Achievement-level results are presented 
in terms of geography achievement levels 



as authorized by the NAEP legislation and 
adopted by the National Assessment 
Governing Board (NAGB). 7 For each 
grade tested, NAGB has adopted three 
achievement levels: Basic , Proficient , and 
Advanced. For reporting purposes, the 
achievement-level cut scores are placed on 
the geography scale, resulting in four 
ranges: below Basic , Basic , Proficient , and 
Advanced. 

The Setting of 
Achievement Levels 

The 1988 NAEP legislation that created 
the National Assessment Governing Board 
directed the Board to identify “appropriate 
achievement goals... for each subject area” 
that NAEP measures. 8 The 2001 NAEP 
reauthorization reaffirmed many of the 
Boards statutory responsibilities, including 
developing “appropriate student achieve- 
ment levels for each grade or age in each 
subject area to be tested . . . ” 9 To follow 
this directive and achieve the mandate of 
the 1988 statute to “improve the form and 
use of NAEP results,” NAGB undertook 
the development of student performance 
standards called “achievement levels.” Since 
1990 the Board has adopted achievement 
levels in mathematics, reading, U.S. history, 
geography, science, writing, and civics. 

The Board defined three levels for each 
grade: Basic , Proficient, and Advanced. The 
Basic level denotes partial mastery of the 
knowledge and skills that are fundamental 
for proficient work at a given grade. The 
Proficient level represents solid academic 



7 No Child Left Behind Act of 2001: Reauthorization of the Elementary and Secondary Education Act. Pub. L. No. 
107-110 (H.R. 1). 

National Assessment of Educational Progress Improvement Act of 1988. Pub. L. No. 100-297, 20, U.S.C. 1211. 

8 National Assessment of Educational Progress Improvement Act of 1988. Pub. L. No. 100-297, 20, U.S.C. 1211. 

9 No Child Left Behind Act of 2001 : Reauthorization of the Elementary and Secondary Education Act. Pub. L. No. 
107-1 10 (H.R. 1). 
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performance. Students reaching this level 
demonstrate competency over challenging 
subject matter. The Advanced level pre- 
sumes mastery of both the Basic and Profi- 
cient levels and superior performance. 

Figure 1.3 presents the policy definitions of 
the achievement levels that apply across all 
grades and subject areas. The policy defini- 
tions guided the development of the 
geography achievement levels, as well as 



Figure 1.3 




Achievement Levels 




Basic 


This level denotes partial mastery of prerequisite knowledge and skills that are 
fundamental for proficient work at each grade. 


Proficient 


This level represents solid academic performance for each grade assessed. Students 
reaching this level have demonstrated competency over challenging subject matter, 
including subject-matter knowledge, application of such knowledge to real-world 
situations, and analytical skills appropriate to the subject matter. 


Advanced 

V 


This level signifies superior performance. 



SOURCE: National Assessment Governing Board. 



the achievement levels established in all 
other subject areas. Adopting three levels 
of achievement for each grade signals the 
importance of looking at more than one 
standard of performance. The Board 
believes, however, that all students should 
reach the Proficient level: the Basic level is 
not the desired goal, but rather represents 
partial mastery that is a step toward Proficient. 



The achievement levels in this report 
were adopted by the Board based on a 
standard-setting process designed and 
conducted under a contract with ACT, Inc. 
To develop these levels, ACT convened a 
cross section of educators and interested 
citizens from across the nation and asked 
them to judge what students should know 
and be able to do relative to a body of 
content reflected in the NAEP framework 
for geography. This achievement-level- 
setting process was reviewed by a variety of 
individuals including policymakers, repre- 
sentatives of professional organizations, 
teachers, parents, and other members of the 
general public. Prior to adopting these 



levels of student achievement, NAGB 
engaged a large number of persons to 
comment on the recommended levels and 
to review the results. 

The results of the achievement-level- 
setting process, after NAGB s approval, 
became a set of achievement-level descrip- 
tions and a set of achievement-level cut 
points on the 0—500 NAEP geography 
scale. The cut points are the scores that 
define the boundaries between below 
Basic , Basic , Proficient , and Advanced perfor- 
mance at grades 4, 8, and 12. The Board 
established these geography achievement 
levels based upon the geography content 
framework. 
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Achievement-Level 
Descriptions for Each Grade 

Specific definitions of the Basic , Proficient , 
and Advanced geography achievement levels 
for grades 4, 8, and 12 are presented in 
figures 1.4 through 1.6. As noted previ- 
ously, the achievement levels are cumula- 
tive. Therefore, students performing at the 
Proficient level also display the competencies 
associated with the Basic level, and students 



at the Advanced level also demonstrate the 
skills and knowledge associated with both 
the Basic and the Proficient, levels. For each 
achievement level listed in figures 1.4 
through 1.6, the scale score that corre- 
sponds to the beginning of that level is 
shown in parentheses. For example, in 
figure 1.4 the scale score of 240 corre- 
sponds to the beginning of the grade 4 
Proficient level of achievement. 



Figure 1.4 




Achievement 

Levels 





Basic 

(187) 



Students should be able to use words or diagrams to define basic geography vocabulary; 
identify personal behaviors and perspectives related to the environment, and describe some 
environmental and cultural issues in their community; use visual and technology tools to 
access information; identify major geographic features on maps and globes; be able to read 
and draw simple maps, map keys, and legends; demonstrate how people depend upon, use, 
and adapt to the environment; and give examples of the movement of people, goods, services, 
and ideas from one place to another. In addition to demonstrating an understanding of how 
individuals are alike and different, they should demonstrate a knowledge of the ways people 
depend on each other. 



Proficient 

(240) 



Students should be able to use fundamental geographic knowledge and vocabulary to identify 
basic geographic patterns and processes; describe an environmental or cultural issue from 
more than one perspective; and read and interpret information from visual and technological 
tools such as photograph maps and globes, aerial photography, and satellite images. They 
should be able to use number and letter grids to plot specific locations; understand relative 
location terms; and sketch simple maps and describe and/or draw landscapes they have 
observed or studied. Proficient students should be able to illustrate how people depend upon, 
adapt to, and modify the environment, describe and/or illustrate geographic aspects of a 
region using fundamental geographic vocabulary and give reasons for current human 
migration; discuss the impact a location has upon cultural similarities and differences; and 
be able to demonstrate how an event in one location can have an impact upon another 
location. 



Advanced 

(276) | 






Students should be able to use basic geographic knowledge and vocabulary to describe global 
patterns and processes; describe ways individuals can protect and enhance environmental 
quality; describe how modifications to the environment may have a variety of consequences; 
explain differing perspectives that apply to local environmental or cultural issues; and 
demonstrate an understanding of forces that result in migration, changing demographics, 
and boundary changes. They should be able to solve simple problems by applying information 
learned through working with visual and technological tools such as aerial and other 
photographs, maps and globes, atlases, news media, and computers. They should be able to 
construct models and sketch and label maps of their own state, the United States, and the 
world; use them to describe and compare differences, similarities, and patterns of change in 
landscapes; and be able to predict the impact a change one location can have on another. 
They should be able to analyze the ways individuals and groups interact. 



SOURCE: National Assessment Governing Board, Geography Framework for the 1994 and 200} National Assessment of Educational Progress. 



Q 0 CHAPTER 1 

ERIC 



GEOGRAPHY REPORT CARD 



26 




Figure 1.5 

Achievement 

Levels 




Basic | Students should possess fundamental knowledge and vocabulary of concepts relating to 
( 242 ) patterns, relationships, distance, directions, scale, boundary, site, and situation; solve 

fundamental locational questions using latitude and longitude; interpret simple map scales; 
identify continents and their physical features, oceans, and various cities; respond accurately 
to descriptive questions using information obtained by use of visual and technological tools 
such as geographic models and/or translate that information into words; explain differences 
between maps and globes; and find a wide range of information using an atlas or almanac. 
Students should be able to recognize and illustrate the relationships that exist between 
humans and their environments, and provide evidence showing how physical habitat can 
influence human activity. They should be able to define a region and identify its distinguishing 
characteristics. Finally, they should be able to demonstrate how the interaction that takes place 
between and among regions is related to the movement of people, goods, services, and ideas. 



Proficient Students should possess a fundamental geographic vocabulary; understand geography’s 
( 282 ) analytical concepts; solve locational questions requiring integration of information from two 
or more sources, such as atlases or globes; compare information presented at different 
scales; and identify a wide variety of physical and cultural features and describe regional 
patterns. Students should be able to respond accurately to interpretive questions using 
geography's visual and technological tools and translate that information into patterns; 
identify differences in map projections and select proper projections for various purposes-, and 
develop a case study working with geography's analytical concepts. In addition, students 
should be able to describe the physical and cultural characteristics of places-, explain how 
places change due to human activity; and explain and illustrate how the concept of regions 
can be used as a strategy for organizing and understanding Earth's surface. Students should 
be able to analyze and interpret data bases and case studies, as well as use information from 
maps to describe the role that regions play in influencing trade and migration patterns and 
cultural and political interaction. 



Advanced Students should have a command of extensive geographic knowledge, analytical concepts, 

( 315 ) and vocabulary; be able to analyze spatial phenomena using a variety of sources with 

information presented at a variety of scales and show relationships between them; and use 
case studies for special analysis and to develop maps and other graphics. Students should be 
able to identify patterns of climate, vegetation, and population across Earth's surface and 
interpret relationships between and among these patterns, and use one category of a map or 
aerial photograph to predict other features of a place such as vegetation based on climate or 
population density based on topographic features. Students should also be able to relate the 
concept of region to specific places and explain how regions change over time due to a variety 
of factors. They should be able to profile a region of their own design using geographic 
concepts, tools, and skills. 



SOURCE: National Assessment Governing Board, Geography Framework for the 1994 and 2001 National Assessment of Educational Progress. 
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Figure 1.6 

Achievement 

Levels 



Basic | Students should possess a knowledge of concepts and terms commonly used in physical and 
(270) I human geography as well as skills enabling them to employ applicable units of measurement 
| and scale when solving simple locational problems using maps and globes. They should be 
J able to read maps; provide examples of plains, plateaus, hills, and mountains; and locate 
! continents, major bodies of water, and selected countries and cities. They should be able to 
| interpret geographic data and use visual and technological tools such as charts, tables, 
j cartograms, and graphs; know the nature of and be able to identify several basic types of 
j map projection; understand the basic physical structure of the planet; explain and apply 
concepts such as continental drift and plate tectonics; and describe geography's analytical 
concepts using case studies. Students should have a comprehensive understanding of spatial 
relationships including the ability to recognize patterns that exist across Earth in terms of 
phenomena, including climate regions, time zones, population distributions, availability of 
resources, vegetation zones, and transportation and communication networks. They should be 
able to develop data bases about specific places and provide a simple analysis about their 
importance. 



Proficient j 

(305) | 



Students should have an extensive understanding and knowledge of the concepts and 
terminology of physical and human geography. They should be able to use geographic 
concepts to analyze spatial phenomena and to discuss economic, political, and social factors 
that define and interpret space. They should be able to do this through the interpretation of 
maps and other visual and technological tools, through the analysis of case studies, the 
utilization of data bases, and the selection of appropriate research materials. Students 
should be able to design their own maps based on descriptive data; describe the physical and 
cultural attributes of major world regions; relate the spatial distribution of population to 
economic and environmental factors; and report both historical and contemporary events 
within a geographic framework usingtools such as special purpose maps, and primary and 
secondary source materials. 



V 



Advanced 

(339) 



Students should possess a comprehensive understanding of geographic knowledge and 
concepts; apply this knowledge to case studies; formulate hypotheses and test geographic 
models that demonstrate complex relationships between physical and human phenomena; 
apply a wide range of map skills; develop maps using fundamental cartographic principles 
including translating narratives about places and events into graphic representations, and 
use other visual and technological tools to perform locational analysis and interpret spatial 
relationships. Students should also be ableto undertake sophisticated analysis from aerial 
photographs or satellite imagery and other visuals. Advanced students should be ableto 
develop criteria assessing issues relating to human spatial organization and environmental 
stability and, through research skills and the application of critical thinking strategies, 
identify alternative solutions. They should be able to compile data bases from disparate 
pieces of information and from these data bases develop generalizations and speculations 
about outcomes when data change. 



SOURCE: National Assessment Governing Board, Geography Framework for the 1994 and 2001 National Assessment of Educational Progress. 
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